Practice of FFmpeg front-end video synthesis

Development background of video synthesis capability

To develop an application with video synthesis capabilities, there is a certain level of complexity in both the principle and application aspects. In principle, video synthesis requires the application of various algorithms to encode and decode audio and video data, and to process the encapsulation of various different audio and video formats; In terms of application, the video synthesis process is relatively long and requires parallel processing of multiple input files to achieve functions such as video filtering, editing, and splicing, making the application scenarios more complex.



The representative of video synthesis applications are various video editing software, which used to mainly exist in the form of native applications. In recent years, with the continuous opening of browser interfaces and capabilities, there have gradually been solutions and approaches for web-based video synthesis capabilities.



This article introduces a video synthesis capability developed based on FFmpeg+WebAssembly. Compared with existing solutions in the community, this solution uses JSON to describe the video synthesis process, which can improve the convenience and flexibility of business side use and correspond to more video synthesis business scenarios.



In the first half of 2023, the trend of AIGC based content creation emerged. The team I am in is responsible for the creation, submission, and other business of Bilibili. During this period, I also participated in related AIGC creation tool projects and was responsible for the development of web front-end video synthesis capabilities in the project.



Technical selection

If audio and video related capabilities need to be introduced into applications, one of the common solutions in the industry is to use FFmpeg. FFmpeg is a well-known audio and video comprehensive processing framework written in C language, which can provide various functions such as audio and video recording, format conversion, editing and synthesis, and streaming.



In order to be able to use FFmpeg in browsers, we need two technologies: WebAssembly+Enscrien:



WebAssembly is a type of assembly language that browsers can run, commonly used in high-performance computing scenarios on the browser side. Assembly languages are generally difficult to write by hand, so there are solutions to compile into WebAssembly through other high-level languages (C/C++, Go, Rust, etc.).

Emscripten is a compilation toolkit suitable for C/C++projects, which can be used to compile C/C++projects into WebAssembly and port them to browsers for runtime. WebAssembly and Emscripten both build an environment for C language projects to run in a browser. In addition, with the actual audio and video processing capabilities provided by the FFmpeg module, theoretically we can perform video synthesis in a browser.

Compile FFmpeg to WebAssembly

To compile FFmpeg into WebAssembly through Enscrien, you need to use Enscrien. Emscripten itself is a collective term for a series of compilation tools, which emulate the classification methods of compilers, linkers, assemblers, and other programs in gcc, and implement the processing of wasm32 object files